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Summary 

This  paper  presents  available  virtual  reality  technology 
as  well  as  technology  that  is  projected  to  become 
available  to  NATO  in  the  near  future.  Areas  discussed 
are  new  PC  technology  (graphics  rendering  and  wearable 
computers),  personal  and  large-volume  displays,  large 
volume  tracking,  force  feedback  interfaces,  and  software 
toolkits.  PCs  presently  render  millions  of  polygons/sec. 
Their  reduced  cost  makes  possible  the  distribution  of 
virtual  environments  at  many  sites  and  in  many 
countries.  Large-volume  displays  are  more  expensive, 
but  allow  more  natural  user  interactions.  They  do  require 
large-volume  tracking  that  is  fast  and  accurate.  Haptic 
interfaces  are  a recent  class  of  input/output  devices  that 
increase  simulation  realism  by  adding  the  sense  of  touch. 
This  comes  at  a cost  of  more  computing  power  and 
better  physical  modeling.  The  modeling  and  program- 
ming needs  of  virtual  reality  are  met  by  software  toolkits 
designed  for  such  simulations. 

1.  Introduction 

Virtual  reality  technology  has  experienced  significant 
advances  in  the  late  nineties,  and  now  has  many 
characteristics  that  may  be  exploited  by  the  military. 
Virtual  reality  has  the  potential  to  significantly  reduce 
training  costs  and  the  risk  to  him.  It  also  has  the  potential 
to  reduce  team  training  costs,  allowing  multi-national 
organizations,  such  as  NATO,  to  have  a unified  training 
system,  without  a unique  training  location.  Virtual 
reality,  as  a computerized  training  environment,  allows 
transparent  gathering  of  data,  and  the  remote  access  to 
such  data,  at  a much  smaller  time  interval,  and  resolution 
than  allowed  by  manual  data  collection  methods.  For  all 
these  reasons  it  is  important  to  inform  the  military 
decision-makers  of  what  technology  and  methods  are 
available  today,  or  what  will  become  available  in  the 
near  future. 

This  report  is  based  on  the  keynote  address  given  by  the 
author  at  the  NATO  Workshop  that  took  place  in  April 
2000  in  Hague.  Then,  as  now,  the  time  and  space 
available  for  such  a review  are  limited.  When  trying  to 
condense  all  this  material,  which  can  easily  take  a 
Semester  to  teach  in  college,  certain  things  had  to  be 
omitted.  Thus  the  present  review  does  not  cover 
networked  communication  as  it  applies  to  shared  VR, 


nor  does  it  cover  human  factor  trials  of  VR  technology. 
Such  topics  are  covered  in  companion  papers.  Emphasis 
here  is  on  commercial  off-the  shelf  technology,  or 
technology  that  is  close  to  commercialization.  Many 
deserving  research  projects  arc  omitted  here,  as  a matter 
of  practicality.  The  interested  reader  who  wants  more 
information  on  such  research  should  consult  the  open 
literature,  such  as  the  Proceedings  of  the  IEEE  Virtual 
Reality  Conference  series  (formerly  VRAIS),  and  other 
such  publications. 

Section  2 of  this  report  presents  significant  changes  in 
the  computing  platforms  that  arc  (or  may  be)  used  in 
VR.  Section  3 describes  the  displays  that  output  the 
graphics  scene  to  the  user,  whether  such  displays  are 
personal  or  large-volume.  Large-volume  displays,  in 
turn,  require  large-volume  trackers,  which  are  the  subject 
of  section  4.  Section  5 presents  the  newer  haptic  inter- 
faces, which  bring  more  realism  to  the  simulation  by 
allowing  the  user  to  touch  and  feel  virtual  objects.  The 
modeling  libraries  needed  by  modern  VR  simulations 
(including  haptics)  are  detailed  in  section  6.  Section  7 
concludes  this  report. 

2.  The  PC  Revolution 

Probably  one  of  the  most  important  changes  that  has 
influenced  the  VR  arena  in  recent  years  is  the 
tremendous  increase  in  PC -based  graphics  rendering 
speed.  The  closing  gap  between  inexpensive  PC -based 
graphics  and  the  lngh-end  SGI  engines  is  clearly 
illustrated  by  Figure  1 . 

The  measure  of  performance  used  for  comparison  here  is 
the  number  of  polygons  rendered  by  the  computer  in  unit 
time.  When  dividing  this  number  by  the  scene  com- 
plexity, one  obtains  the  screen  refresh  rate  in  frames/ 
second  (how  many  snapshots  of  the  virtual  scene  the 
computer  can  render  per  unit  time).  The  more  complex 
the  scene,  the  less  frames/second,  which  in  turn  can 
result  in  a disturbing  saccadic  graphics  [Burdea  & 
Coiffet,  1994], 


1 Based  on  the  author’s  presentation  at  RTA'HFM  Workshop  007,  The  Hague,  Netherlands,  13-15  April.  © Grigore  C.  Burdea, 
except  for  certain  illustrations. 


Paper  presented  at  the  RIO  HFM  Workshop  on  “What  Is  Essential  for  Virtual  Reality  Systems  to  Meet  Military 
Human  Performance  Goals?  ”,  held  in  The  Hague,  The  Netherlands,  13-15  April  2000,  and  published  in  RTO  MP-058. 
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Figure  1:  SGI  graphics  vs.  PC-based  graphics 


In  1994  a 486  processor  PC  with  SPEA  FIRE  board  was 
capable  of  7,000  polygons/sec.  A modem  Pentium  III  PC 
with  Wildcat  graphics  board  can  do  6,000,000  polygons/ 
sec,  and  costs  only  6,000  dollars  or  so.  During  the  same 
time  the  performance  of  high-end  graphics  workstations 
produced  by  SGI  rose  from  300,000  polygons/sec.  on  a 
Reality  Engine  in  1994  to  13,000,000  polygons/sec. 
today  on  a multi-pipe  Infinite  Reality  2 [Real  Time 
Graphics,  2000].  While  its  performance  is  twice  that  of 
the  fastest  PC  rendering  board,  its  price  is  two  to  three 
hundred  thousand  dollars,  which  makes  it  affordable  to 
only  a few!  By  significantly  improving  performance, 
while  actually  reducing  costs  in  the  late  nineties,  the  PC 
industry  made  possible  the  much-desired  widespread  use 
of  desktop  3-D  graphics. 

The  second  important  change  in  the  computer  industry  is 
the  tendency  to  miniaturize  the  computer,  to  the  point 
that  it  becomes  wearable  on  the  user.  Figure  2 shows  just 
such  an  example,  namely  the  Mobile  Assistant  IV R pro- 
duced by  Xybemaut  Co.  (Fairfax  VA,  USA).  It  consists 
of  a CPU  unit  with  a Pentium  processor  and  simplified 
keyboard,  a head-mounted  display,  a microphone  for 
voice  input,  and  a camera  worn  on  the  user’s  head.  By 
coupling  this  with  wireless  communication,  the  user  gets 
freedom  of  motion  within  the  range  of  the  wireless 
transmitter,  and  as  a function  of  battery  life. 

User  freedom  of  motion  is  very  important  to  the  VR 
application  designer,  because  it  increases  the  naturalness 
of  the  interaction,  and  thus  the  feeling  of  immersion  that 
the  user  has.  At  the  present  time  the  Mobile  Assistant 
docs  not  have  sufficient  computing  power  to  incorporate 
graphics  real-time  rendering.  Such  a capability  is 
expected  to  appear  in  subsequent  models  of  the  device. 


Figure  2:  Mobile  Assistant  IVk  wearable  computer. 
Courtesy  of  CAIP  Center,  Rutgers  University. 
Reprinted  by  permission 


3.  Graphic  Displays 

Another  important  component  of  VR  systems  are  the 
graphics  displays,  which  present  the  computer,  rendered 
scene  to  the  user.  Such  displays  may  be  classified  as 
personal  displays,  for  a single  user,  and  large-volume 
displays,  which  allow  several  users  to  view  the  same 
scene  in  a given  location.  Both  types  of  displays  have 
advanced  significantly  in  the  past  decade,  as  will  be 
described  next. 


3.1  Personal  displays 

The  most  prevalent  type  of  personal  display  available  in 
the  nineties  were  head-mounted  displays  (HMDs),  which 
projected  the  image  close  to  the  user’s  head.  Early 
HMDs  were  very  bulky  and  heavy,  weighing  over  two 
kilograms  in  the  case  of  the  VPL  “Eyephone.”  Their 
resolution  was  poor  (360x240  pixels)  ow'ing  to  the  LCD 
technology  of  the  time.  Compared  to  this,  modern 
HMDs,  such  as  the  SONY  Glasstron 1 shown  in  Figure  3, 
have  an  SVGA  resolution  (832x624  pixels).  The 
improvement  in  image  resolution  was  coupled  with  a 
dramatic  reduction  in  weight  (120  grams  for  the 
Glasstron).  Unfortunately,  the  necessary  miniaturization 
means  that  the  user’s  field  of  view  (FOV)  is  small 
(30x22  degrees)  compared  to  the  Eyephone  FOV  of 
90x60  degrees.  Recently  SONY  has  announced  it  will 
stop  producing  Glasstrons.  Its  logical  replacement  is  the 
Olympus  Eye-Trek  HMD  (37x22  degrees)  weighing  a 
little  over  100  grams  [Olympus,  2000], 
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Figure  3:  The  SONY  Glasstron  Courtesy  of  InterSense 
Co.  Reprinted  by  permission 

The  user’s  natural  field  of  view  is  180  degrees  horizontal 
and  almost  as  much  vertical.  The  human  vision  system, 
unlike  the  HMDs,  has  an  uneven  resolution  over  its 
FOV.  The  highest  resolution  is  in  a central  “foveating 
area,”  while  the  retina  has  much  lower  resolution  away 
from  the  foveating  area.  By  rendering  the  image  at 
constant  resolution  the  computer  essentially  wastes 
pixels,  since  the  eye  cannot  see  them.  Eye  trackers  allow 
computers  to  detect  where  the  user  focuses  on  an  image. 
It  is  then  possible  to  render  the  corresponding  virtual 
scene  in  high  resolution,  and  the  rest  of  the  scene  in 
lower  resolution.  A review  on  the  state-of-the-art  in  eye 
tracking  can  be  found  in  [Isdale,  2000],  Figure  4 shows 
an  HMD  retrofitted  with  an  eye  tracker. 


Figure  4:  The  SONY  Glasstron  fitted  with  an  eye 
tracker.  Courtesy  of  VR  News.  Reprinted  by  permission 

Military  reconnaissance  training  applications  can  benefit 
from  a “customized”  HMD,  such  as  the  V8  Binoculars 
(Virtual  Research  Systems  Inc.,  Santa  Clara  CA,  USA) 
shown  in  Figure  5.  These  binoculars  integrate  dual  LCD 
displays,  with  VGA  resolution,  and  a FOV  of  up  to  60 
degrees.  Its  optics  allows  individual  focus  adjustment, 
and  its  weight  is  680  grams.  By  integrating  a position 
tracker  (discussed  later  in  this  report),  the  computer 


senses  the  3-D  aim  of  the  binoculars  and  displays  the 
corresponding  scene  in  real  time. 


Figure  5:  The  V8  Binoculars  HMD.  Courtesy  of 
Virtual  Research  Systems  Inc.  Reprinted  by  permission 

Other  types  of  graphics  displays,  available  today,  are 
“virtual  window's”  and  auto-stereoscopic  displays.  The 
WindowVR®  produced  by  Virtual  Research  Systems 
Inc.,  is  shown  in  Figure  6.  In  has  a flat-panel  display  (a 
touch-sensitive  display  in  some  versions)  with  handles 
and  suspension  cable.  A tracker  inside  the  display  allows 
the  computer  to  change  the  scene  and  give  the  user  the 
sensation  of  looking  at  a virtual  w'orld  through  a 
window.  Buttons  on  the  handles  allow  actions  and 
navigation  within  the  VR  simulation. 


Figure  6:  The  WindowVRK  . Courtesy  of  Virtual 
Research  Systems  Inc.  Reprinted  by  permission 


Auto-stereoscopic  workstations,  such  as  the  ones 
produced  by  Dimension  Technologies  Inc.  (Rochester 
NY,  USA),  use  backlighting  of  a flat  panel  to  produce  a 
stereo  image.  As  seen  in  Figure  7,  the  image  appears  to 
float  in  space,  without  the  need  for  HMDs.  Its  resolution 
is  1280x1024,  which  is  superior  to  that  of  LCD-based 
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displays  [Dimension  Technologies  Inc.,  2000]. 
Unfortunately,  the  stereo  image  can  be  seen  from  only  a 
small  viewing  volume  and  the  brightness  of  the  image 
suffers  owing  to  the  lighting  scheme  used.  Thus  graphics 
appears  dim  when  compared  HMDs  or  active  glasses 
(discussed  later  in  this  report). 


Figure  7:  An  auto-stereoscopic  workstation.  Courtesy 
of  DTI  Inc.  Reprinted  by  permission 

3.2  Large-volume  displays 

Large-volume  displays  offer  a much  larger  stereo 
viewing  area,  high  resolution,  and  a way  for  many 
participants  to  view  and  interact  with  the  same  virtual 
scene.  One  class  of  large-volume  displays  is  "virtual 
workbenches,”  such  as  the  one  shown  in  Figure  8.  It  uses 
a CRT  projector  and  mirrors  to  “place”  the  stereo  scene 
on  top  of  its  table.  The  integration  of  its  projector  within 
the  display  table  makes  for  a compact  design,  and  the 
tilting  mechanism  can  change  the  user’s  viewing  cone. 
The  Baron  can  tilt  from  fully  horizontal  to  fully  vertical, 
which  transforms  it  essentially  in  a “virtual  wall”  type 
display.  Future  designs  will  replace  the  CRT  technology 
with  much  brighter  digital  mirror  technology.  Then  it 
will  be  possible  to  use  such  displays  without  having  to 
reduce  the  room  ambient  lighting  level. 


Figure  8:  The  BARCO  Baron"  3-D  display.  Courtesy 
of  BARCO  Co.  Reprinted  by  pennission 


Figure  9 shows  a marine  amphibious  landing  exercise 
scene  produced  by  a workbench-type  display  [Ilix  et  al., 
1999],  The  usual  2-D  military  symbols  were  replaced  by 
3-D  icons  of  trucks,  airplanes,  ships,  etc.,  shown  on  a 3- 
D terrain  map.  Such  a scene  is  much  easier  to 
comprehend,  and  may  reduce  errors  in  a high  stress 
combat  situation.  Furthermore,  the  use  of  3-D  icons 
coupled  with  haptics  (not  used  in  this  particular  training 
scenario)  opens  the  way  for  a different  kind  of  C&C 
interaction. 


Figure  9:  Sea  Dragon  Marine  landing  exercise. 
Courtesy  of  the  Naval  Research  Laboratory, 
Washington  DC.  Reprinted  by  permission 


Using  a haptic  glove  (discussed  later  in  this  report)  the 
military  commander  may  then  be  able  to  grasp  and  feel 
such  3-D  objects.  The  force  feedback  addition  to  the 
simulation  has  at  least  two  important  advantages  for  the 
military  decision-maker.  First,  he  knows  he  has  complete 
and  unique  control  over  the  unit  whose  symbol  he 
grasped.  This  is  true  even  if  he  momentarily  looks  away 
from  the  screen.  Second,  the  hardness  of  the  symbol  can 
give  him  valuable  information  on  the  unit’s  state  of 
readiness/strength  level.  A tank  3-D  icon  that  feels  soft 
may  indicate  that  unit  is  at  half  strength,  due  to  losses.  A 
tanker  plane  that  feels  hard  may  indicate  that  it  is  full  of 
fuel.  etc. 

An  example  of  a C&C  application  using  a haptic  glove  is 
the  system  demonstrated  by  the  CATP  Center  at  Rutgers 
University,  and  shown  in  Figure  10  [Medl  et  al.  1998].  It 
consists  of  a distributed  architecture,  with  a multi-modal 
interface.  The  user  gives  voice  commands  that  are 
detected  by  a microphone  array  placed  on  top  of  a PC. 
He  can  select  and  move  military  symbols  on  a map  using 
either  an  eye  tracker,  or  a force  feedback  glove  (Rutgers 
Master  glove  [Burdea,  1996]).  The  New  Jersey  National 
Guard,  with  little  prior  training,  tested  the  system 
successfully  in  1997. 
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Figure  10:  Multi-modal  interface  C&C  exercise. 
Courtesy  of  the  CAIP  Center,  Rutgers  University. 
Reprinted  by  permission 


A larger  type  of  display  than  the  workbench  is  the 
CAVE®  stereo  display  made  by  Fakespace  Systems 
(Ontario,  Canada).  As  shown  in  Figure  11,  the  CAVE 
consists  of  multiple  wall-type  displays  assembled  in  a 
cube  geometry.  Each  wall  has  its  own  CRT  projector, 
driven  by  a separate  graphics  pipe  of  a multi-processor 
high-end  SGI  or  equivalent  computer.  The  user  enters 
the  CAVE  and  is  looking  at  the  display  walls  through 
“active”  stereo  glasses,  such  as  those  shown  in  Figure 
12.  Infrared  emitters  located  in  the  corners  of  the  CAVE 
control  the  opening  and  closing  of  shutters  incorporated 
in  the  stereo  glasses.  They  alternately  block  the  view  of 
each  eye,  which  allows  the  brain  to  register  the  two 
images  rendered  by  the  computer  separately  and  create 
the  stereo  effect. 

With  his  FOV  filled  by  the  graphics  the  CAVE  user  feels 
immersed  in  the  virtual  world.  Furthermore,  the  work 
volume  in  which  the  user  sees  stereo  and  can  interact 
with  virtual  “floating”  objects  is  much  larger  than  for  a 
workbench.  These  advantages  come  at  a price,  as  the 
cost  of  the  CAVE  is  five  times  that  of  a workbench 
display.  T this  is  added  the  cost  of  the  high-end  graphics 
computer,  bringing  the  system  close  to  one  million 
dollars  at  the  time  of  this  writing. 


Figure  11:  The  CAVE  stereo  display.  Courtesy  of 
Fakespace  Systems  Inc.  Reprinted  by  permission 


Figure  12:  Stereo  “active”  glasses  fitted  with  the 
InterSense  tracker.  Courtesy  of  InterSense  Co. 
Reprinted  by  permission 


Recently  Fakespace  Systems  introduced  the  “Re- 
configurable  Advanced  Visualization  Environment” 
(RAVE)  shown  in  Figure  13.  Unlike  the  CAVE,  which 
has  a fixed  geometry,  RAVE  can  change  its 
configuration  depending  on  the  user’s  needs.  Thus  its  3 
m x 2.9  m x 3.7  m modules  can  be  assembled  to  form  a 
straight  wall  geometry,  where  three  display  units  are 
side-to-side.  Other  available  configurations  include  a u- 
shape,  or  a cube  (CAVE-type  geometry).  Alternately,  it 
can  separate  itself  into  two  half-cube  independent 
displays.  As  expected,  the  cost  of  RAVE  surpasses  that 
of  the  CAVE. 


Figure  13:  The  RAVE  re-configurable  stereo  display. 
Courtesy  of  Fakespace  Systems  Inc.  Reprinted  by 
permission 


4.  Large-Volume  Tracking 

The  user’s  ability  to  see  graphics  that  fill  most  of  his 
FOV  is  a good  start  towards  a more  immersive  virtual 
environment.  Another  important  requirement  is  to  allow 
the  user  to  interact  with  virtual  objects  he  sees.  Thus  the 
computer  needs  to  know  as  accurately  as  possible  the 
current  3-D  position  of  the  user’s  hand(s),  head,  or 
whole  body  within  this  large  working  volume. 

4.1  Magnetic  tracking  errors 

Computers  determine  the  user’s  position  by  interpreting 
data  fed  by  3-D  trackers  worn  on  the  body.  The 
overwhelming  majority  of  today’s  trackers  are 
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electromagnetic  ones,  consisting  of  a stationary  source  of 
pulsating  magnetic  fields,  one  to  several  receivers  (coils) 
worn  by  the  user,  and  an  electronic  control  box.  The 
voltages  induced  in  the  receivers  are  transformed  in 
absolute  position/orientation  values  by  the  control  box, 
and  then  sent  to  the  computer  running  the  simulation. 

An  example  of  high-end  magnetic  tracker  is  the 
MotionStar®  wireless  tracking  suit  produced  by 
Ascension  Technology  Co.  (Burlington  VT,  USA), 
shown  in  Figure  14.  The  suit  incorporates  20  magnetic 
tracker  receivers  placed  at  critical  locations  on  the  user’s 
body,  such  as  the  wrist,  ankle,  hip,  etc.  The  receivers  are 
wired  and  the  electronic  control/communication  box 
worn  on  a backpack.  Owing  to  its  own  power  supply  (a 
battery  with  two-hour  life),  the  suit  can  work 
independently  and  furnish  up  to  100  readings/sec.  within 
three  meters  from  the  tracker  source.  Such  a range  would 
accommodate  two  RAVE  modules,  if  placed  side-by- 
side,  with  the  source  centrally  located. 


Figure  14:  The  MotionStar®  wireless  tracking  suit. 
Courtesy  of  Ascension  Technology  Co.  Reprinted  by 
pennission 

There  is  however  a problem  with  all  magnetic  trackers, 
which  affects  their  accuracy.  This  is  due  to  interference 
from  other  magnetic  fields,  or  from  metallic  objects. 
Such  problems  were  reported  with  the  MotionStar® 
[Marcus,  1997],  but  also  with  the  Polhemus 
LongRanger®  (Colchester  VT,  USA)  [Trefftz  & Burdea, 
2000].  Figure  15  shows  the  magnitude  of  the  error  vector 
for  a LongRanger®  installed  on  a wooden  tripod  in  the 


Human-Machine  Laboratory  at  Rutgers  University.  The 
tripod  allowed  the  height  of  the  tracker  source  to  be 
varied,  while  precise  position  of  the  receiver  was 
measured  mechanically.  The  errors  grew  geometrically 
with  the  distance  from  the  tracker  source,  as  expected. 
However,  errors  also  varied  depending  on  the  source 
height  above  the  floor.  The  most  accurate  measurements 
were  obtained  when  the  source  was  at  1 .68  m above  the 
floor.  Errors  grew  when  the  source  was  too  close  to 
either  the  ceiling  or  to  the  floor,  owing  to  the  metallic 
beams  used  in  the  laboratory  room  construction. 
Additional  experimental  measurements  showed  that  the 
metal  in  the  large-volume  display  (in  this  case  a BARCO 
Baron  workbench)  introduced  more  tracking  errors. 

Magnitude  of  Error  Vector  / Moving  Tripod 


35.0 

-or- 

a 

5 30  0 

c 

fc  25.0 

§ 

> 20.0 

fc 

£ 15.0 

o 

* 100 

3 

E>  50 


IJ  Source  2.02  m above  floor 

n 



\ r 

Source  1 .68  m above  floor 

■V  — 
* : 

Source  1.37  m 

\ « 

• \ ‘ ■ 

I above  floor 

\ * ! 

Si- 


ml  3 2^ 


20  30  40  50  60  70 

Transmitter-Receiver  Dista nee  (inches) 


Figure  15:  The  Polhemus  LongRanger  tracking  errors 
[Trefftz  & Burdea,  2000] 


The  above  findings,  and  those  of  others,  point  out  the 
inadequacy  of  magnetic  trackers  when  working  in 
typical  large-volume  display  environments.  Thus  one  is 
left  with  two  alternatives.  The  first  is  to  build  a special 
structure,  designed  from  the  start  to  house  large-volume 
displays  and  the  related  trackers,  and  to  redesign  the 
display  to  reduce  the  amount  of  metal.  The  second,  and 
an  easier  alternative,  is  to  change  the  tracker. 

4.2  Tnertial/ultrasonic  trackers 

In  recent  years  a new  generation  of  trackers  has  become 
commercially  available.  These  arc  hybrid  3-D  position 
trackers,  such  as  the  1S-600  shown  in  Figure  16, 
manufactured  by  InterSense  Inc.  (Burlington  MA,  USA). 
They  use  a combination  of  inertial  and  ultrasonic  sensing 
technology,  with  the  inertial  component  used  for  position 
measurements,  and  the  ultrasonic  component  used  to 
provide  a zero  position  and  to  collect  for  drift.  One  or 
more  inertial  cubes  arc  placed  on  the  user,  or  on  his 
interface,  together  with  sonic  disks  (as  previously  shown 
in  Figure  12  for  active  glasses).  The  inertial  cube  signal 
is  read  by  an  electronic  box,  which  also  drives  ultrasonic 
receivers  placed  on  the  ceiling  in  a cross  configuration. 
Since  these  trackers  do  not  use  magnetic  fields,  they  are 
immune  to  the  type  of  interference  associated  with 
magnetic  trackers. 
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Figure  16:  The  InterSense  IS-600®  inertial/ultrasonic 
tracker.  Courtesy  of  InterSense  Co.  Reprinted  by 
permission 


A recent  addition  to  the  InterSense  tracking  family  is  the 
IS-900  LAT  (large-area-tracker)  [InterSense,  2000],  It 
can  extend  its6mx6m><3m  standard  tracking  volume 
to  a maximum  tracking  area  of  900  m:  using  up  to  24 
expansion  hubs.  Its  measurement  accuracy,  resolution 
and  latency  are  better  than  for  magnetic  trackers. 

5.  Haptic  Interfaces 

Another  important  change  taking  place  in  current  VR 
technology  is  the  addition  of  haptic  feedback,  namely 
tactile  and  force  feedback.  Tactile  feedback  gives  the 
user  the  ability  to  touch  and  feel  the  smoothness  of 
virtual  object  surfaces,  their  temperature,  slippage,  and 
contact  surface  geometry.  Force  feedback  conveys 
information  on  object  weight,  inertia,  mechanical 
compliance,  degree  of  mobility,  viscosity,  etc.  The 
addition  of  haptic  feedback  clearly  increases  simulation 
realism  in  general.  Furthermore,  haptic  feedback  allows 
object  manipulation  in  occluded,  foggy  or  dark  virtual 
environments,  a task  that  would  otherwise  be  difficult  or 
even  impossible  to  complete. 

5.1  General-purpose  haptic  interfaces 

Haptic  interfaces  may  be  classified  as  general-purpose 
ones,  which  can  be  used  for  many  tasks  (including 
military  ones),  and  special-purpose  haptic  interfaces, 
which  are  designed  specifically  for  military  applications. 
An  example  of  a general-purpose  force  feedback 
interface  is  the  PHANToM®  arm  Desktop  produced  by 
SensAble  Technologies  Co.  (Woburn  MA,  USA),  and 
shown  in  Figure  17.  The  interface  measures  the  position 
and  orientation  of  the  stylus  1000  times/ sec,  and  applies 
forces  of  up  to  10  N to  the  user’s  hand  in  response  to 
actions  in  the  virtual  environment.  The  high  bandwidth 
of  the  PHANToM  allows  it  to  combine  force  with  tactile 
feedback,  such  that  the  roughness  or  stickiness  of  a 
surface  can  be  simulated  as  well. 

A typical  application  developed  for  the  PHANToM  is 
“digital  sculpting,”  as  illustrated  in  Figure  17.  The  user 
is  presented  with  a block  of  “digital  clay,”  which  he 
deforms,  sculpts,  polishes,  using  the  stylus.  The  user 
feels  the  resistance  of  the  material,  as  well  as  the 
influence  of  the  change  in  virtual  tool  to  which  the  stylus 
is  mapped. 


Figure  17:  The  PHANToM'  desktop  force  feedback 
arm.  Courtesy  of  SensAble  Co.  Reprinted  by  permission 

Once  the  3-D  model  is  sculpted,  its  files  can  be 
downloaded  lo  a NC  mill  or  similar  equipment,  to  build 
an  actual  prototype.  This  is  also  applicable  to  the  weapon 
design  cycle,  speeding  up  its  mock-up  phase. 

Another  use  of  the  PHANToM  is  in  mine  detection 
training,  an  application  being  currently  developed  by  the 
French  Ministry  of  Defense  (see  companion  paper  by 
Todeschini).  The  force  feedback  arm  integrated  with  this 
system  is  designed  to  replicate  the  tactile  sensation  the 
trainee  uses  to  detect  a mine.  Since  in  actual  operations 
such  a task  must  have  a 100%  rate  of  success,  it  is  clear 
that  a realistic  trainer  should  be  useful.  The  difficulty  in 
realizing  such  a system  is  to  realistically  replicate  the 
dynamic  force  “signature”  associated  with  various  mines 
and  ground  conditions. 


Figure  18:  Digital  sculpting  with  force  feedback. 
Courtesy  of  SensAble  Co.  Reprinted  by  permission 


One  drawback  of  the  PHANToM  arm  is  that  it  is  not 
able  to  provide  finger-specific  forces,  such  as  those 
present  in  dexterous  tasks,  when  contact  is  at  the 
fingertip.  Such  tasks  could  be  assembly  training, 
servicing  of  military  hardware,  or  training  in  explosive 
handling.  For  such  instances  a better  haptic  interface  is  a 
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force  feedback  glove,  such  as  the  CyberGrasp"  glove 
produced  by  Virtual  Technologies  Inc.  (Palo  Alto  CA, 
USA),  shown  in  Figure  19. 


Figure  19:  The  CyberGrasp  glove  in  a CyberPack 
configuration.  Courtesy  of  Virtual  Technologies  Co. 
Reprinted  by  permission 


The  glove  consists  of  a CyberGlove  [Kramer  et  ah, 
1991]  used  for  position  measurements  on  which  is 
retrofitted  a force  feedback  exoskeleton  driven  by  cables. 
The  tendons  are  routed  to  an  electronic  control  box 
housing  electrical  actuators  and  communication 
hardware.  The  force  output  is  about  16  N per  finger, 
which  is  larger  than  the  PPIANToM  output.  Unlike  the 
PHANToM,  which  sits  on  a desk,  and  limits  freedom  of 
motion,  the  CyberGrasp  glove  is  worn.  Furthermore,  the 
CyberPack"  configuration  places  the  control  box  in  a 
backpack,  such  that  the  user  can  walk  around  and  grasp 
objects  and  feel  their  hardness.  Its  limiting  factors  then 
are  weight,  (which  can  lead  to  user  fatigue)  and  the 
range  of  the  tracker  measuring  wnst  3-D  position. 

Another  limitation  of  the  CyberGrasp  haptic  glove  is  the 
lack  of  force  feedback  to  the  wrist.  Thus  grasped  objects 
seem  weightless,  with  no  inertia  and  no  mechanical 
restraints.  Recently  Virtual  Technology  announced  the 
CyberForce®  haptic  interface  shown  in  Figure  20.  It 
consists  of  a six  degrees-of-freedom  force  feedback  arm 
connected  to  the  back  palm.  By  combining  wrist  force 
feedback  with  the  force  feedback  glove,  the  ability  to 
simulate  weight  and  inertia  are  added  while  the  user 
preserves  his  hand  dexterity  [Kramer,  2000]. 
Furthermore,  there  is  no  need  for  a wrist  position  tracker, 
since  the  force  feedback  arm  measures  wrist  position 
faster  and  without  metallic  interference.  Unfortunately, 
the  dimensions  of  the  arm  limit  the  user’s  freedom  of 
motion.  Furthermore,  the  overall  system  control  becomes 


much  more  complex,  which  may  lead  to  system 
instabilities. 


Figure  20:  The  CyberGrasp  glove  in  a CyberForce 
configuration.  Courtesy  of  Virtual  Technologies  Co. 
Reprinted  by  permission 


In  certain  military  applications  of  VR,  such  as  infantry 
training,  there  is  a need  to  simulate  running,  or  walking 
uphill,  or  through  uneven  terrain.  In  such  cases  haptic 
feedback  to  the  body  becomes  important  in  order  to  have 
realistic  training.  One  system  that  addresses  these  needs 
has  been  recently  developed  by  Sarcos  Co  (Salt  Lake 
IJT,  USA)  and  the  University  oflJtah  [Hollerbach  et  ah, 
1999].  As  shown  in  Figure  21,  the  user  is  located  in  front 
of  a three-wall  display  filling  most  of  his  FOV  and 
stands  on  a treadmill.  By  tracking  his  walking/runmng 
on  the  treadmill,  the  computer  updates  the  virtual  scene 
accordingly.  A force  feedback  arm  is  attached  to  the 
user's  torso  through  a harness.  The  arm  applies  resistive 
and  inertial  forces  to  simulate  uneven  terrain  and  other 
effects.  A rope  attached  to  the  ceiling  prevents  injury  in 
case  of  tripping  and  falling. 


Figure  21:  The  treadport  VR  system.  Courtesy  of 
University  of  Utah  CS  Dept.  Reprinted  by  pennission 


Recently,  Japanese  researchers  proposed  the  replacement 
of  the  treadmill  approach  -with  an  “active  floor”,  as 
shown  in  Figure  22  [Noma  et  ah,  2000].  The  floor  is 
composed  of  modular  actuator  tiles  that  can  change  slope 
under  computer  control.  The  user’s  motion  is  tracked  by 
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a vision  system,  and  the  tiles  actuated  as  needed  to 
replicate  uneven  terrain.  Thus,  unlike  the  walking-in- 
place paradigm  of  treadmill  systems,  the  active  floor 
approach  allows  natural  walking  over  the  whole  surface 
of  the  floor.  There  is  no  need  for  a force  feedback  arm 
attached  to  the  user’s  back,  and  no  need  for  a safety 
rope.  The  limitation  in  this  case  is  the  size  and  amount  of 
slope  that  can  be  produced  by  the  active  tiles. 


Figure  22:  The  active  floor  VR  system  [Noma  et  al. 
2000].  © IEEE.  Reprinted  by  permission 


5.2  Special-purpose  haptic  interfaces 

All  the  haptic  interfaces  presented  so  far  are  general- 
purpose,  since  they  can  be  used  in  military  applications 
but  were  not  specifically  designed  for  such.  By  contrast, 
special-purpose  haptic  interfaces  are  designed  from  the 
start  to  provide  force/touch  feedback  to  military  VR 
tasks.  An  example  is  the  Stinger  trainer  prototype 
developed  at  TNO  (The  Hague,  The  Netherlands)  [Jense, 
1993],  shown  in  Figure  23.  It  consists  of  a plastic  mock- 
up  of  the  missile  launcher,  which  is  instrumented  to  track 
the  user’s  aim,  and  to  sense  when  switches  are 
depressed.  Furthermore,  a virtual  environment  showing 
the  enemy  aircraft  is  presented  to  the  trainee  on  an 
HMD.  The  advantage  of  this  system  is  that  a much  more 
compact  set-up  replaces  the  classical  large-dome  training 
system.  Furthermore,  all  user  actions  are  stored 
transparently  and  his  performance  data  is  available  on 
the  computer.  The  force  feedback  sensation  is  produced 
naturally  by  the  plastic  mock-up,  without  need  for  more 
expensive  (and  heavier)  hardware.  The  system  is  now 
being  used  in  training  the  German  Air  Force,  as 
described  in  the  companion  paper  by  Reichert. 

Another  example  of  special-purpose  haptics  is  the  anti- 
tank missile  trainer  system  recently  developed  by  the 
Fifth  Dimension  Technologies  Co.  (Pretoria,  South 
Africa),  which  is  shown  in  Figure  24.  It  uses  a mock-up 
of  the  rocket  launcher,  similar  to  the  TNO  Stinger 
trainer,  which  provides  direct  tactile  feedback.  Other 
similarities  include  the  used  of  a HMD  to  display  the 
virtual  battlefield  to  the  trainee,  and  a 3-D  tracker  to 
determine  his  direction  of  view. 


Figure  23:  The  Stinger  VR  training  prototype  Courtesy 
of  TNO,  The  Netherlands.  Reprinted  by  permission 


Figure  24:  The  anti-tank  VR  training  prototype 
Courtesy  of  5DT  Co.,  Pretoria,  South  Africa.  Reprinted 
by  permission 

Another  type  of  special-purpose  haptic  interface  is  the 
parachute-training  simulator  developed  by  Systems 
Technology  Inc.  (Hawthorne  CA,  USA).  As  shown  in 
Figure  25,  the  system  uses  a full-size  parachute  harness, 
and  an  HMD  showing  a detailed  3-D  jump  scene  (insert). 
The  scene  moves  in  response  to  either  head  motion,  or 
the  toggle  of  the  parachute  harness  [Systems  Technology 
Inc.  2000],  Wind  effects  are  added,  to  train  the  jumper  in 
coping  with  adverse  landing  conditions.  Playback  of  user 
actions  and  instructor  actions  are  used  to  help  acquire  the 
necessary  skills. 
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Figure  25:  The  VR  parachute  training  system.  Courtesy 
of  Systems  Technology  Inc.  Reprinted  by  permission 

6.  Modeling  Tools 

So  far  this  report  has  reviewed  the  computing  hardware 
and  the  interfaces  available  to  develop  VR  applications. 
The  third  element  needed  is  a VR  toolkit,  i.c.  software 
libraries  specifically  developed  for  programming  virtual 
environments.  Such  toolkits  offer  certain  advantages  to 
the  developer,  namely  drivers  for  most  VR  I/O  devices, 
certain  3-D  graphics  routines,  ease  of  portability,  etc.  In 
turn  VR  toolkits  can  be  classified  as  general-purpose  and 
special-purpose  libraries. 

6.1  General-purpose  Modeling  Tools 

The  most  used  VR  programming  toolkit  today,  by  far,  is 
“WorldToolKit”  (WTK),  produced  by  Sense8,  a division 
of  Engineering  Animation  Inc.  (Ames  IA,  USA).  It 
consists  of  over  1000  C/C++  object-oriented  functions, 
which  are  executed,  in  an  infinite  loop  during  the 
simulation.  An  example  of  a scene  created  with  WTK  is 
the  tank  interior  simulation  shown  in  Figure  26.  By 
importing  CAD  files,  doing  smooth  shaded  graphics, 
textured  surfaces,  dynamic  effects,  WTK  allows  very 
realistic  simulations  to  be  created. 

Another  facility'  provided  hy  WTK  (in  its  “World-up” 
version)  is  graphics  programming,  as  shown  in  Figure 
27.  Thus  the  kinematics  dependencies  and  other  virtual 
object  characteristics  can  be  easily  specified  using  a 
scene  graph.  At  run  time  the  software  goes  through  the 
nodes  of  this  scene  graph. 

For  all  its  advantages  WTK  has  at  least  two 
disadvantages,  namely  cost  and  short-lived  releases.  The 
license  cost  for  WTK  is  an  order  of  magnitude  more  than 
for  widespread  PC  software,  reflecting  the  small  market 
for  VR  products.  This  is  aggravated  by  numerous 
releases,  which  many  times  are  not  compatible  with 
earlier  ones.  As  such  a military  application  developed 


with  an  earlier  release  may  not  run  when  the  library  is 
updated  (currently  WTK  is  at  release  9). 


Figure  26:  The  tank  interior  created  with  WTK. 
Courtesy  of  EAI  Co.  Reprinted  by  permission 
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Figure  27:  The  World-up  graph  scene.  Courtesy  of  EAI 
Co.  Reprinted  by  permission 


A 3-D  programming  toolkit  which  is  free  is  Java3D 
produced  by  Sun  Microsystems  (Palo  Alto  CA,  USA). 
Java3D  programming  is  also  based  on  a scene  graph. 
However,  the  software  is  still  under  development,  and 
certain  drawbacks  exist,  when  compared  with  WTK. 
One  of  the  most  important  limitations  of  Java3D  is  its 
inability  to  deliver  a uniform  rendering  speed,  as 
uncovered  by  recent  tests  done  at  Rutgers  University. 
Figure  26  [Boian,  2000]  shows  the  same  scene  being 
rendered  on  a dual-processor  450  MHz  Pentium  PC, 
using  (a)  WTK  (release  8)  and  (b)  Java  3D  (release 
1.1.2).  The  scene  consisted  of  40,000  textured  polygons, 
and  collision  detection  was  activated.  When  WTK  was 
used,  the  average  time  to  render  one  frame  was  123  ms 
(8.1  frames/sec),  with  a standard  deviation  of  about  10 
ms.  Interestingly  enough,  Java3D  was  37%  faster,  with 
an  average  rendering  speed  of  11.1  frames/sec.  Its 
average  time  to  render  a frame  was  only  90  ms. 
Unfortunately,  its  standard  deviation  was  84  ms,  or 
840%  larger  that  for  WTK. 
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Figure  28:  Comparison  of  frame  rendering  speed  and 
consistency  between:  a)  WTK;  b)  Java3D  [Boian, 
2000],  Reprinted  by  permission 


Generalizations  can  be  risky,  and  certainly  SUN 
Microsystems  will  address  some  of  these  drawbacks  in 
newer  Java3D  releases.  However  such  large  standard 
deviations  in  frame  rendering  time,  as  present  in  the 
current  Java3D  release  will  adversely  impact  interactions 
in  the  virtual  environment,  especially  where  force 
feedback  is  concerned. 

Force  feedback  calculation  is  preceded  by  a collision 
detection  step  that  is  used  by  the  computer  to  determine 
if  there  is  interaction  in  the  virtual  environment.  Such  an 
algorithm  needs  to  be  both  accurate  and  fast,  which  is 
difficult  in  complex  virtual  environments.  One  example 
is  CAD  analysis  for  accessibility.  Complex  assemblies, 
such  as  “crowded”  aircraft  engines,  are  difficult  to 
design  and  even  more  difficult  to  service.  Researchers  at 
Boeing  Co.  (Seattle  WA,  USA)  have  developed  the 
“voxel  point  shell”  (VPS)  method  of  collision  detection 
to  cope  for  such  application  needs  [McNeely  et  al., 
1999].  VPS  builds  a point  shell  around  the  surface  of  a 
single  moving  object  in  a pre-computing  stage.  At  run 
time,  this  point  shell  is  checked  for  collision  with  the 
static  environment,  and  the  resulting  force/torque  applied 
to  the  user.  Tests  done  using  a complex  model  of  a 
Boeing  777  with  almost  600  thousand  polygons,  shown 
in  Figure  29,  allowed  haptic  rendering  at  a constant  rate 


of  1000  Hz.  The  visual  frame  rate  was  20  frames/sec, 
using  Boeing’s  proprietary  “FlyThru”  rendering 
software. 


b 

Figure  29: 


6.2  Special-purpose  modeling  toolkits 

Special-purpose  toolkits  have  been  developed  to  help 
certain  types  of  simulations.  For  example,  Virtual 
Technologies  have  introduced  the  VirtualHand®  Suite 
2000,  which  is  a library'  designed  to  work  with  the 
CyberGlove,  CyberGrasp,  and  CyberTouch  interfaces 
[Virtual  Technologies,  2000].  It  helps  develop 
applications  where  interaction  with  the  objects  is  at  the 
level  of  the  hand,  and  includes  collision  detection,  a 
force  feedback  API  and  networking  capabilities. 

Another  special-purpose  toolkit  is  the  GHOST  library 
developed  by  SensAble  Technologies  for  their 
PHANToM  arm.  It  allows  the  mixing  of  scene  graph  and 
direct  force  field  programming,  in  scenes  with 
complexities  up  to  250.000  polygons  (mesh 
configuration).  Multiple  PHANToM  Desktop  models 
can  be  supported  in  a daisy-chain  arrangement  on  a 
single  host  communication  port. 

Finally,  the  DI-Guy  library  developed  by  Boston 
Dynamics  (Cambridge  MA,  USA)  helps  program 
simulations  involving  dismounted  infantry,  special 
operations  and  peacekeeping  operation  tasks  by 
providing  an  intelligent-agent  based  library  [Boston 
Dynamics  Inc.,  1997].  As  can  be  seen  in  Figure  30,  the 
toolkit  allows  users  to  control  avatars  that  respond  to 
real-time  task-level  control.  Once  they  are  given 
behavior  (walk,  kneel,  crawl,  etc.)  and  travel  parameters, 
they  execute  the  action  through  motion  interpolation. 
This  allowrs  multiple  DI-Guy  characters  to  be  included  in 
a given  virtual  scene.  The  toolkit  is  currently  supported 
by  WTK  (Release  9)  and  by  Vega  (Paradigm 
Simulations  Inc.,  Dallas  TX,  USA).  Vega  LynX  allows  a 
point-and-click  interaction  environment. 
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Figure  30:  Scene  created  with  the  DI-Guy  toolkit  for 
dismounted  infantry  training.  Courtesy  of  Boston 
Dynamics  Tnc.  Reprinted  by  permission 


7.  Conclusions 

There  is  no  doubt  that  VR  technology  has  been  going 
through  a rapid  change.  A major  impact  on  the 
widespread  use  of  this  technology  in  the  military  and 
other  areas  is  the  tremendous  decrease  in  computer 
prices,  and  increase  in  PC -based  graphics  speed.  The 
miniaturization  of  the  PC  in  its  present  form  allows  for 
portability',  which  results  in  increased  user  freedom  of 
motion  and  simulation  realism.  Large-volume  displays 
arc  also  adding  to  the  user  ability  to  interact  with  large 
simulation  volumes.  New  trackers  have  overcome  the 
limitation  of  magnetic  technology  and  can  be  used  for 
wide  area  tracking  and  interaction.  Portable  haptic 
interfaces  also  add  to  realism,  especially  in  tasks 
involving  manual  dexterity.  Programming  toolkits  now 
offer  a complex  programming  environment  integrating 
the  various  modalities  of  interacting  with  the  virtual 
world.  All  these  developments  point  to  more  useful 
military  application  of  VR,  primarily  in  training,  but  also 
in  C&C  and  weapon  design/prototyping.  Human  factor 
studies  need  to  validate  the  technology  and  its 
usefulness. 
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